Feature selection discards columns. Feature extraction does something more interesting: it builds a small set of new features as linear combinations of the originals, chosen so that as much of the data's variance as possible survives the reduction. Principal Component Analysis is the canonical method.
We build PCA from its mathematical foundations — the covariance matrix, eigenvalues and eigenvectors, eigendecomposition — then state the algorithm as a learning phase (fit on training data only) and an application phase. The explained variance ratio and the scree plot give us a principled way to choose how many components to keep. We finish with a full hand-computed example on the Iris dataset and a comparative case study on Adult Census Income that pits filter selection, wrapper selection and PCA against each other on the same problem.
The curse of dimensionality presents several challenges:
PCA is a dimensionality reduction technique that transforms data into a new coordinate system:
Applications:
PCA aims to find the directions of maximum variance in high-dimensional data and projects the data onto a new subspace with equal or fewer dimensions than the original one.
Key Insight: The orthogonal axes (principal components) of the new subspace can be interpreted as the directions of maximum variance given the constraint that the new feature axes are orthogonal to each other.
Mathematical Note: If vectors \(a\) and \(b\) are orthogonal, then \(a \cdot b = 0\)
Eigenvalues and eigenvectors are fundamental mathematical concepts used in PCA:
In PCA context, eigenvalues and eigenvectors have special properties:
PCA consists of two phases: a Learning Phase and an Application Phase. The distinction matters in practice, because the parameters PCA learns — the feature means, the standard deviations and the projection matrix \(W\) — must be computed on the training set only. They are then applied unchanged to the validation and test data.
We construct a \(d \times k\) transformation matrix \(W\) that maps original features to reduced dimensions:
Example: 13-dimensional wine data → 3-dimensional PCA space
Key insight: W contains the selected principal components as its columns. This is the point at which the eigenvalues and eigenvectors stop being an abstract property of the covariance matrix and become a concrete dimensionality reduction step.
Before working through the mathematics in detail, it is worth stating the properties that make the transformed features useful:
Eigendecomposition is at the mathematical core of PCA.
The covariance matrix is a special type of square matrix: it's symmetric, meaning the matrix equals its transpose (\(A = A^T\)).
When we decompose a symmetric matrix, we get valuable properties:
Key insight: Eigenvalues and eigenvectors come in pairs: each eigenvalue has a corresponding eigenvector that shows the direction of variance.
Most important: The eigenvector with the largest eigenvalue points in the direction of maximum variance in the dataset—this becomes our first principal component.
The covariance matrix is a symmetric \(d \times d\) matrix where \(d\) is the number of features in the dataset. It stores pairwise covariances between all features, showing how each pair of features varies together.
Where \(\mu_j\) and \(\mu_k\) are the sample means of features j and k respectively. Note that the sample means are zero if we standardized the dataset.
For a 3-feature dataset, the covariance matrix \(\Sigma\) looks like:
Important: The diagonal contains variances (\(\sigma_1^2, \sigma_2^2, \sigma_3^2\)), while off-diagonal elements are covariances (\(\sigma_{12}, \sigma_{13}, \sigma_{23}\)).
Each principal component explains a portion of the total variance in the dataset. The explained variance helps determine how many principal components to keep.
Total Variance: The sum of variances of all original features equals the sum of variances of all principal components.
Explained Variance Ratio: The proportion of variance explained by each principal component.
The previous section introduced explained variance informally. We now write it out in terms of the eigenvalues, which is the form used in practice to choose \(k\).
Each eigenvalue \(\lambda_j\) quantifies the variance captured by its corresponding principal component. The proportion of total variance explained by component \(j\) is:
The cumulative EVR tells us what fraction of information is retained when we keep the first \(k\) components:
Let us compute PCA on the first 8 Iris samples. Because all 4 features are measured in cm with similar ranges, we only center (we do not divide by std for pedagogical clarity).
| Sample | Sepal L | Sepal W | Petal L | Petal W |
|---|---|---|---|---|
| 1 | 5.1 | 3.5 | 1.4 | 0.2 |
| 2 | 4.9 | 3.0 | 1.4 | 0.2 |
| 3 | 4.7 | 3.2 | 1.3 | 0.2 |
| 4 | 4.6 | 3.1 | 1.5 | 0.2 |
| 5 | 5.0 | 3.6 | 1.4 | 0.2 |
| 6 | 5.4 | 3.9 | 1.7 | 0.4 |
| 7 | 4.6 | 3.4 | 1.4 | 0.3 |
| 8 | 5.0 | 3.4 | 1.5 | 0.2 |
Covariance Eigendecomposition: \(\Sigma = Q\Lambda Q^\top\), producing eigenvalues:
We skipped full standardization here (cm features with similar scales). Always use full standardization when:
We compare four feature-engineering pipelines on the Adult Income dataset (predict income > $50K / year, ~30K rows, 14 mixed features). Evaluation metric: AUC.
| Pipeline | # Features Used | Training AUC | Test AUC | Training Time |
|---|---|---|---|---|
| Baseline (all features) | 14 | 0.921 | 0.884 | 1.0× (ref) |
| Filter (χ² + ANOVA) | 9 | 0.910 | 0.881 | 0.8× |
| Wrapper (Fwd/Bwd + KNN) | 7 | 0.932 | 0.876 | 12.4× |
| PCA (k = 6 components) | 6 | 0.914 | 0.888 | 1.3× |
Consider a 2D dataset with correlated features:
The Wine dataset eigenvalues (sorted) are: \([4.84,\; 2.42,\; 1.55,\; 0.96,\; 0.84,\; 0.66,\; 0.52,\; 0.35,\; 0.31,\; 0.21,\; 0.18,\; 0.15,\; 0.11]\). Click to reveal answers.
(a) What percentage of variance is explained by the first component alone?
(b) What is the cumulative variance explained by the first 3 components?
(c) A rule-of-thumb says "keep components until ≥ 95 % variance". Roughly how many components would that be for Wine?
Back to the Adult Census case study. Five students interpret the results. Who is correct?
| Student | Claim |
|---|---|
| Ali | Wrapper is best because its training AUC is highest (0.932). |
| Bilal | PCA is best because its test AUC is highest (0.888). |
| Chloe | Filter is best because it's fastest. |
| Dua | Wrapper is overfitting because test AUC (0.876) < train AUC (0.932). |
Given a dataset with the following covariance matrix:
Step 1: Find eigenvalues
Step 2: Find eigenvectors
Step 3: Principal Components
A 6-feature dataset produces sorted eigenvalues: \(\lambda = [3.0,\; 2.0,\; 1.0,\; 0.6,\; 0.3,\; 0.1]\).
Step 1: Total variance = \(\sum \lambda_i = 3.0 + 2.0 + 1.0 + 0.6 + 0.3 + 0.1 = 7.0\)
Step 2: Individual EVRs:
Step 3: How many PCs to keep for ≥ 90 % cumulative variance?
4 components give 94.3 % ≥ 90 %. So \(k = 4\).
Step 4: Compression ratio = \(\frac{k}{d} = \frac{4}{6} \approx 67\%\) of the original feature count.
Given the covariance matrix:
Tasks:
You have a dataset with two features:
Task: What happens if you apply PCA without standardizing the data first? Which feature will dominate the first principal component?
Problem: Without standardization, PCA will be dominated by the feature with the largest scale (Income).
Why: PCA maximizes variance. Income has much larger absolute values and thus larger variance in the original scale, even though the relative variability (coefficient of variation) might be similar.
Result: The first principal component will be heavily weighted toward Income, and Age will have minimal influence.
Solution: Always standardize features (mean=0, variance=1) before applying PCA when features have different scales.
After applying PCA to a dataset with 10 features, you get the following eigenvalues for the principal components:
[4.5, 3.2, 1.8, 0.9, 0.6, 0.3, 0.2, 0.1, 0.05, 0.05]
Tasks:
Sorted eigenvalues from a 5-feature PCA run: \([2.5,\; 1.5,\; 0.7,\; 0.2,\; 0.1]\).
Answer all 6 questions. Click an option for instant feedback.
Your score: 0 / 6